Information Fusion in Attention Networks Using Adaptive and Multi-Level Factorized Bilinear Pooling for Audio-Visual Emotion Recognition

Multimodal emotion recognition is a challenging task in computing as it quite difficult to extract discriminative features identify the subtle differences human emotions with abstract concept and multiple expressions. Moreover, how fully utilize both audio visual information still an open problem. In this paper, we propose novel multimodal fusion attention network for audio-visual based on adaptive multi-level factorized bilinear pooling (FBP). First, stream, convolutional (FCN) equipped 1-D mechanism local response normalization designed speech recognition. Next, global FBP (G-FBP) approach presented perform by integrating self-attention video stream proposed stream. To improve G-FBP, strategy (AG-FBP) dynamically calculate weight of two modalities devised emotion-related representation vectors from respective modalities. Finally, information, (AM-FBP) introduced combining global-trunk intra-trunk data one recording top AG-FBP. Tested IEMOCAP corpus only new FCN method outperforms state-of-the-art results accuracy 71.40%. validated AFEW database EmotiW2019 sub-challenge recognition, AM-FBP achieves best 63.09% 75.49% respectively test set.